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Abstract Body 

Limit 4 pages single-spaced. 


Background / Context: 

Description of prior research and its intellectual context. 

Mastering basic numeracy and literacy skills is one of the most fundamental goals of education. 
However, it is estimated that 250 million primary-school-age children lack basic reading, writing 
and math skills (UN, 2013). Children living in war and poverty stricken countries are among the 
least likely to attain those basic goals. The United States Agency for International Development 
(USAID) has set out to improve basic literacy -and to a lesser extent, basic numeracy — skills 
for 100 million children in low-income countries by 2015. USAID hopes to accomplish this 
through the development of innovative teaching techniques and learning materials, maximizing 
time spent on instruction, and strengthening household, community, and school partnerships (see 
http ://www.usaid. gov/what- we-do/ education) . The present study is part of an impact evaluation 
of one such effort in the Democratic Republic of the Congo (DRC). 

Purpose / Objective / Research Question / Focus of Study: 

Description of the focus of the research. 

The main purpose of the present study is to describe and discuss results from a cluster 
randomized trial of the “Opportunities for Equitable Access to Quality Basic Education” (OPEQ) 
program. OPEQ was developed by the International Rescue Committee (IRC) with funding from 
USAID, to better the quality of teaching and learning processes and enhance children and 
youth’s academic and socio-emotional outcomes in the DRC. The presentation will focus on 
intent-to-treat impacts after one year of implementation. 

Setting: 

Description of the research location. 

The study took place in Katanga, an eastern province of the DRC. As of 2013, the DRC ranks 
second to last in the human development index, an indicator of wellbeing that measures health, 
education and income (http://hdr.undp.org/en/statistics/) . The state of education is particularly 
dismal: since 1986, public financing for education has been subjected to substantial cuts, leaving 
the educational system at the mercy of household resources. Today, education expenditures 
comprise only 2.5% of the GDP, the school life expectancy is 8 years, and there is a 66% literacy 
rate among those 1 5 and older ( https://www.cia.gov/library/publications/the-world- 
factbook/geos/cg.html) . The lack of educational resources severely curtails the life-opportunities 
of Congolese children and youth, as well as hinders overall social progress in the DRC. 

Population / Participants / Subjects: 

Description of the participants in the study: who, how many, key features, or characteristics. 

The study uses data from the first and second years of a three-year cluster randomized trial of 
OPEQ. Two hundred and five (205) schools located in six educational subdivisions (i.e., 
Kasenga, Kongolo, Kalemie, Mutshasha, Lubudi, Kambove) in the province of Katanga were 
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invited and agreed to be recipients of OPEQ. Schools were organized in 54 clusters of 2 to 6 
schools to facilitate service delivery. For the purposes of the experimental evaluation, 84 schools 
were randomly selected to take part in data collection. One or two schools were randomly 
selected from each of the 54 clusters. One school was selected when the cluster had 3 schools or 
less, and 2 were selected when the cluster had more than 3 schools. 

At baseline, schools had an average of 379 students (SD: 226; min: 82, max: 1,130) and eight 
classrooms (SD: 2.83; min: 5, max: 16). The majority of schools were Catholic (39.3%) or 
Protestant (32.1%), but other religious affiliations included Orthodox (7.1%), Kimbanguiste 
(2.4%) and Muslim (1.2%). On average, 74% of teachers per school were male; 91.7% of 
schools had no electricity, 3.6% had no roof, 16.7% had no latrines, and 28.6% had no benches. 

Students in second through fourth grades were randomly selected to participate in the evaluation. 
In the second year, 17 to 83 students per school were assessed. The effective sample for this 
study consists of 8,879 students (29.6% second grade, 27.9% third grade, 27.2% fourth grade) 
with a valid demographic survey, and scores on the math or reading assessments. Boys comprise 
51.5% of the sample; the majority of the children reported Swahili as their main language (70%), 
and on average children were 10.4 years old (SD = 2.0). 

Intervention / Program / Practice: 

Description of the intervention, program, or practice, including details of administration and duration. 

OPEQ aims to enhance teachers’ motivation, the quality of school settings and teaching practices, 
and children’s academic achievement and socio-emotional wellbeing. The intervention has two 
primary and interrelated components. First, an innovative curriculum which integrates high quality 
reading and math lessons with IRC's Healing Classrooms, a protocol of techniques to create safe and 
inclusive learning environments for all learners, is built into a teacher training package. Second, a 
school-based collaborative professional development system of continuous in-service teacher training 
and coaching is implemented. The structure is based on an historical practice of the DRC’s 
educational system: the Forums of Pedagogical Exchange (FPE). FPE’s consist of teacher-learning 
circles that are designed to meet: weekly at grade level; monthly at school level; and quarterly at 
school cluster (2-6 schools) level. FPEs enable teachers to collaboratively explore their practices, 
brainstorm solutions to challenges and identify and celebrate successes. These services are delivered 
by Master Trainers (MT; one per cluster of 2 to 6 schools) composed of teachers, headmasters, 
pedagogical advisors, inspectors and key technical staff from the Ministry of Education. 

Research Design: 

Description of the research design. 

Randomization to treatment took place within each of the subdivisions. Clusters were assigned to 
one of three treatment conditions: a Pilot cohort that started receiving the intervention in 201 1; a 
group of clusters that started receiving the intervention in 2012 (Cohort 1); and a group which began 
the intervention in 2013 (Cohort 2). Intent-to-treat estimates after one year of the pilot intervention 
will be obtained by comparing schools in clusters assigned to the Pilot Cohort to schools in clusters 
assigned to Cohorts 1 and 2. 
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Data Collection and Analysis: 

Description of the methods for collecting and analyzing data. 

Data collection. Data were collected once per year over three years. Assent and consent were 
requested from all children at the time of data collection and refusal to participate was very rare. 
Parental consent could not be obtained due to logistical challenges that would have prohibited 
conducting the evaluation. Instead, the Ministry of Education and IRC’s field team widely 
advertised the evaluation in each school and community to ensure that parents were fully informed 
and had the opportunity to ask any questions, raise any concerns, and opt out. High levels of student 
mobility and logistical difficulties impeded following the same cohort of children over the course 
of the study; therefore, different children were assessed at each wave. Analyses for the present 
study focus exclusively on students with data in the second year. Baseline data are used to adjust 
for school-level pre-treatment characteristics. 

In order to reduce participant burden and gather a rich amount of information, students were 
administered a demographic and a socio-emotional wellbeing survey, and were randomly assigned to 
complete a math or reading assessment. The full protocol took no more than 45 minutes per child. 

Local staff trained by the OPEQ team were in charge of data collection. Surveys and assessments 
were conducted in French (the official language of instruction), but Swahili (the most common local 
language) was used for the demographic and socio-emotional wellbeing surveys, and for instructions 
in the math assessment when children had difficulties understanding French. 

The focus of the current paper is on children’s scores in the Early Grade Math Assessment (EGMA, 
RTI, 2009) and Early Grade Reading Assessment (EGRA, RTI, 2009b). 

Analysis. Intent-to-treat impacts will be estimated by fitting two-level multilevel models (HLM 6.02 
and 7, Raudenbush & Bryck, 2002). Multilevel modeling adjusts for the nested structure of the data 
(i.e., students nested in schools). Independent models will be fitted for children’s math and reading 
scores. Unconditional models will be fitted first, to estimate the amount of variance attributed to 
differences between schools and between children within schools. Subsequent models will estimate 
intent-to-treat effects after one year of the intervention. 

All models will adjust for school-level geographical location (i.e., subdivision dummies), cluster size 
(i.e., a binary variable comparing clusters where one vs. two schools were sampled), child gender, 
and age. Baseline school-level math and reading scores will be adjusted for to examine whether 
intent-to-treat impacts are attenuated as a function of schools’ pre-treatment status. Cross-level 
interactions will be explored to test for heterogeneity of effects. Model fit will be assessed by 
inspecting deviance statistics using the chi-squared distribution. 

Findings / Results: 

Description of the main findings with specific details. 

Descriptive status for key children’s baseline demographic characteristics and reading and math 
performance are summarized in tables 1 through 3. Students were randomized to complete 


SREE Spring 2014 Conference Abstract 


3 



4 


different sets of assessments; therefore, the effective sample for which descriptives are presented 
consists of 4,965 children who completed the demographic survey and were administered the Early 
Grade Reading Assessment (EGRA; RTI, 2009) or Early Grade Math Assessment (EGMA, RTI, 
2009b). Descriptives for the full sample will be presented at the time of the conference. 

Preliminary analysis of baseline data indicates very low levels of performance in math and 
reading, better performance on math compared to reading, higher scores for older compared to 
younger children, and higher scores for boys compared to girls. Analysis examining the 
associations between children’s academic and socio-emotional outcomes; children’s individual, 
household, and school characteristics; and intent-to-treat effects will be presented at the 
conference. 

Conclusions: 

Description of conclusions, recommendations, and limitations based on findings. 

Baseline descriptive results not only reveal very low levels of performance in both math and reading 
in our sample of 2 nd to 4 lh grade children in Katanga province, but also hint at important gender 
disparities that will be further explored at the time of the conference. In addition, the fact that 
reading scores appear to be lower than math scores, coupled with the low percentage of children who 
reported speaking French at home, suggests that learning of French as a second language may be one 
of the critical challenges faced by Congolese children in schools. Practical strategies to support 
language learning could include earlier, more extensive, and more intentional immersion in French, 
or more instruction (i.e., beyond third grade and/or with more supportive materials) in the mother 
tongue. 

Preliminary results also raise questions about how best to set benchmarks against which to measure 
the success of interventions like OPEQ, in settings in which rigorous evaluations of interventions of 
this nature are rare or poorly documented. For example, OPEQ proposed to improve children’s math 
and reading scores by 30%. However, considering what children gain during the primary school 
years in the absence of interventions, an increase of 30% seems rather ambitious. Alternative ways 
of operationalizing and setting benchmarks to assess the success of programs like OPEQ, as well as 
strategies to increase the probability of producing and detecting substantive changes in children’s 
outcomes, will be discussed at the conference. 

Finally, in addition to the presence and strength of what appear to be important differences in 
performance by grade, gender, and geographic location, there may be large within-group differences 
to be described and understood. Identifying other sources of variation in children's performance, as 
proposed in the current paper, is a critical first step in fine-tuning and strengthening the strategies to 
improve children’s outcomes. 
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Table 1. Percentages of boys and girls per grade and subdivision, baseline 


Grade 


2nd 


3rd 


4th 

Subdivision 

% Boys 

% Girls 

% Boys 

% Girls 

% Boys 

% Girls 

Kambove 

55.06 

44.94 

50.60 

49.40 

58.33 

41.67 

Kasenga 

55.92 

44.08 

55.56 

44.44 

63.98 

36.02 

Kalemie 

50.00 

50.00 

47.69 

52.31 

53.17 

46.83 

Kongolo 

50.92 

49.08 

59.33 

40.67 

58.75 

41.25 

Mutshatsha 

50.48 

49.52 

51.99 

48.01 

59.72 

40.28 

Lubudi 

50.27 

49.73 

54.02 

45.98 

51.00 

49.00 

Average across all 
subdivisions 

52.11 

47.89 

53.20 

46.80 

57.49 

42.51 


Table 2. Percentage of children who speak each language by subdivision, baseline 1 


Language 

Subdivision 

Lingala 

Kiswahili 

Tshiluba 

French 

Other 

Kambove 

0.20 

95.13 

0.78 

1.95 

33.72 

Kasenga 

0.00 

39.51 

0.00 

3.40 

87.04 

Kalemie 

1.90 

99.53 

0.35 

0.47 

14.87 

Kongolo 

0.20 

98.45 

0.72 

0.52 

63.12 

Mutshatsha 

1.10 

99.23 

2.20 

4.30 

43.39 

Lubudi 

1.00 

93.48 

2.79 

5.49 

47.67 

Average across all 
subdivisions 

0.73 

87.55 

1.14 

2.69 

48.30 
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Table 3. Reading and Math scores by gender, grade and subdivision 


EGRA 


Grade 

3 


Gender 

Girls Boys 1 

Mean 


Vocabulary (20) 

Initial Sound Identification (10) 

Knowledge of graphemes (100) 
Familiar word reading (50)* 
Invented word decoding (50)* 
Oral passage reading (50)* 
Reading comprehension (5)* 
Listening Comprehension 

Writing a complete sentence 

(3) ** 

EGMA 

Number Identification (30) 
Quantity Discrimination (10) 
Missing Number (10) 

Addition (24) 

Subtraction (24) 

Multiplication (10) 

Word Problems (6)* 

Geometric Shape Identification 

( 4 ) 

Geometric Shape Naming (4) 


n 


Mean 


2939 

6.56 

8.04 

9.58 

2951 

0.68 

1.04 

1.32 

2205 

11.30 

16.76 

22.99 

1900 

- 

3.44 

6.99 

1906 

- 

2.78 

5.78 

1898 

- 

3.98 

8.75 

1919 

- 

0.16 

0.39 

2951 

0.45 

0.63 

0.90 

940 

- 

- 

0.81 



Grade 



2 

3 

4 

n 


Mean 


2920 

14.19 

12.15 

15.72 

2928 

6.29 

5.56 

6.43 

2928 

2.40 

3.38 

4.07 

2935 

4.82 

7.49 

10.77 

2935 

3.30 

5.03 

7.66 

1900 

- 

0.69 

1.38 

2923 

2.02 

2.35 

3.06 

2936 

2.22 

2.53 

2.72 

2936 

1.24 

1.58 

1.79 


7.69 

8.29 

7.82 

0.89 

1.10 

0.15 

16.20 

18.60 

10.87 

4.22 

5.96 

2.47 

3.45 

4.91 

1.66 

4.79 

7.53 

3.02 

0.22 

0.32 

0.07 

0.61 

0.69 

0.28 

0.65 

0.92 

0.63 

Gender 

Girls 

Boys 

1 

Mean 

13.54 

14.40 

12.16 

5.98 

6.20 

6.73 

3.06 

3.42 

2.66 

7.09 

7.99 

6.75 

4.73 

5.67 

3.54 

0.77 

1.24 

0.56 

2.33 

2.57 

2.23 

2.41 

2.53 

2.65 

1.45 

1.59 

1.34 


Katanga Subdivisions*** 


2 

4 

Mean 

4 

5 

6 

5.69 

8.91 

8.57 

8.31 

8.10 

0.06 

2.90 

1.66 

0.33 

0.49 

8.19 

24.95 

20.41 

20.06 

13.62 

1.72 

10.95 

6.52 

5.21 

2.64 

1.04 

9.39 

5.02 

4.58 

2.28 

1.91 

12.98 

8.95 

6.21 

2.75 

0.04 

0.65 

0.48 

0.19 

0.07 

0.31 

1.03 

1.02 

0.53 

0.53 

0.58 

1.44 

0.67 

0.83 

0.60 

Katanga Subdivisions' 



2 

3 

Mean 

4 

5 

6 

10.23 

18.01 

13.26 

16.05 

12.97 

5.01 

6.43 

5.34 

6.54 

6.53 

2.04 

4.54 

3.04 

3.69 

3.14 

4.95 

8.55 

8.71 

8.94 

6.60 

2.96 

5.87 

6.83 

6.86 

4.09 

0.84 

1.13 

1.47 

1.10 

0.82 

1.29 

3.07 

2.04 

3.31 

2.48 

1.56 

2.74 

2.53 

2.65 

2.57 

1.25 

2.37 

1.39 

1.50 

1.27 


* Sub-test not administered to 2nd graders, ** Sub-test not administered to 2ndor 3rdgraders, *** 1: Kambove, 2: Kasenga, 3: Kalemie, 4: Kongolo, 
5: Mutshatsha, 6: Lubudi 
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